Skip to content

TST-50: Property-based and adversarial input tests#818

Merged
Chris0Jeky merged 15 commits intomainfrom
test/property-based-adversarial-input
Apr 12, 2026
Merged

TST-50: Property-based and adversarial input tests#818
Chris0Jeky merged 15 commits intomainfrom
test/property-based-adversarial-input

Conversation

@Chris0Jeky
Copy link
Copy Markdown
Owner

Summary

  • Adds webhook URL adversarial tests (RFC 3986 edge cases, SSRF vectors, dangerous schemes, credentials in URLs)
  • Adds proposal operations adversarial tests (malformed types, missing fields, extra unknown fields, boundary values)
  • Adds capture endpoint adversarial tests (binary data, null bytes, very long strings, nested JSON, injection payloads)
  • Adds no-500s meta-test sweeping boards, capture, cards, search, columns, labels with randomized content
  • Adds capture provenance serialization round-trip property tests (CaptureProvenanceV1, CapturePayloadV1)
  • Adds SQLite query boundary value tests (GUID formats, DateTime boundaries, string edge cases, position/WipLimit boundaries)
  • Adds frontend adversarial data tests (proposal parameters, provenance round-trip, webhook URL safety, numeric boundaries)
  • Documents two known API bugs: proposal endpoint returns 500 for XSS in actionType and nested parameter JSON (skip-annotated)

Closes #717

Test plan

  • All new property/fuzz tests pass (58 domain, 11 application, 144 API, frontend)
  • No existing tests broken (4010 backend, 1927 frontend all pass)
  • No new NuGet packages added (uses existing FsCheck and fast-check)
  • 2 known bugs documented as skip-annotated tests

Test RFC 3986 edge-case URLs, dangerous schemes (javascript:, data:, file:),
SSRF vectors (localhost, metadata endpoints), URLs with credentials,
malformed URLs, and Unicode in URLs against the webhook endpoint.
All verify NO 500 responses.
Test malformed proposal JSON (wrong types, missing fields, extra unknown
fields, enum out of range), adversarial summary strings, boundary-length
summaries, and expiry minutes boundaries. Documents two known 500 bugs
in skip-annotated tests: XSS in actionType and nested parameter JSON.
Exercise capture/inbox with boundary text lengths, null bytes, control
characters, Unicode edge cases, nested JSON, XSS/injection payloads,
adversarial optional fields, malformed JSON bodies, and empty/nonexistent
board IDs. All verify stored-verbatim semantics and no 500s.
Deterministic-seed random generator sends 100 board, 100 capture, 50 card,
50 search, 50 column, and 50 label requests with random content mixing
ASCII, CJK, Arabic, combining diacriticals, and control characters.
Verifies no 500 responses and no exposed stack traces.
Property-based tests verifying CaptureProvenanceV1, CapturePayloadV1,
and CreateCaptureItemDto survive JSON serialize/deserialize with identity.
Exercises adversarial strings in provenance fields, DateTimeOffset
boundary values, and arbitrary JSON deserialization safety.
Test GUID format variations (empty, uppercase, lowercase, mixed-case),
DateTimeOffset boundaries (min, max, epoch, Y2038, Y2K), string edge
cases (empty, whitespace, max-length, over-max), position boundaries
(negative, zero, MAX_INT), WipLimit boundaries, and entity Touch
property across all entity types.
Test proposal operation parameter round-trips, capture provenance
round-trips with adversarial strings, webhook URL display safety for
dangerous schemes, numeric boundary values (MAX_SAFE_INTEGER, Infinity,
NaN), and capture text with binary-like data and performance bounds.
Remove unused RandomGuidString() method from No500sMetaTest.
Improve CaptureProvenanceArb generator to vary string fields
(promptVersion, provider, model, sourceSurface) across runs
instead of using fixed values.
@Chris0Jeky
Copy link
Copy Markdown
Owner Author

Self-review findings

Reviewed all 7 new test files. Issues found and addressed:

Fixed

  1. Dead code: RandomGuidString() in No500sMetaTest was unused -- removed.
  2. Weak property generation: CaptureProvenanceArb generator used fixed strings ("v1.0", "mock") for PromptVersion, Provider, Model, and SourceSurface fields. Now generates from varied sets so the property test actually exercises different values.

Accepted (not bugs)

  1. Random thread safety: No500sMetaTest.Rng is a static Random with seed 42. Since xunit runs test methods within a class sequentially by default, this is safe. The deterministic seed ensures reproducibility.
  2. Frontend malformedParams test pattern: The inner try/catch in malformed operation parameters string does not throw when parsed cautiously absorbs JSON.parse failures. This is intentional -- the test verifies the try/catch pattern itself does not throw unexpectedly.

Bugs found (pre-existing, not introduced by this PR)

  1. Proposal endpoint returns 500 for operations with XSS in actionType -- documented as KnownBug_XssInActionType_Causes500 (skip-annotated).
  2. Proposal endpoint returns 500 for operations with nested parameter JSON -- documented as KnownBug_DeepNestedParameters_Causes500 (skip-annotated).

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 61924ae6e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

new StringContent("{}", Encoding.UTF8, "application/json"));
}

if ((int)response.StatusCode >= 500)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Fail the meta-test when a 500 response is returned

In AuthenticatedEndpoints_NeverReturn500WithStackTrace, the test only inspects the body if status is 500, but never asserts that 500 is forbidden, so a plain 500 response without stack-trace text still passes. That creates a false negative for the exact regression this meta-test is supposed to catch (NeverReturn500), allowing server errors on these endpoints to slip through CI unnoticed.

Useful? React with 👍 / 👎.

new CreateOutboundWebhookSubscriptionDto(endpointUrl));

// Valid HTTPS URLs should either succeed or be accepted
((int)response.StatusCode).Should().BeLessThan(500);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Assert success for valid webhook URLs

CreateWebhook_WithValidUrl_Succeeds currently checks only status < 500, which means a regression that rejects all valid HTTPS webhook URLs with 4xx would still pass this test. Because this method is the only positive-path check in this file, it should assert a success/acceptance status to actually protect valid URL behavior.

Useful? React with 👍 / 👎.


const elapsed = performance.now() - start
// Should complete well under 1 second
expect(elapsed).toBeLessThan(1000)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove hard wall-clock threshold from unit test

The test enforces elapsed < 1000ms for JSON serialize/parse of a 50K string, which depends on CI machine load and runtime variance rather than functional correctness. On slower or contended runners this can fail intermittently even when behavior is correct, introducing flaky test failures into the suite.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive suite of adversarial and property-based tests to improve the robustness of the API and domain logic. The changes include new test classes for capture, proposal operations, and webhook URL validation, alongside fuzz tests for serialization round-trips and boundary-value tests for domain entities. My feedback highlights concerns regarding thread safety in random number generation, the need for null byte coverage in fuzzing, the removal of unused code, and the optimization of property-based tests by removing unnecessary sleeps.


// ─────────────────────── Random content generators ───────────────────────

private static readonly Random Rng = new(42); // deterministic seed for reproducibility
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The System.Random class is not thread-safe. Since this Rng instance is static and shared across the class, concurrent execution of tests (which can occur in xUnit depending on configuration) may lead to race conditions, resulting in corrupted state or zero values. Consider using lock (Rng) when accessing it, or using a thread-safe alternative if parallelism is expected.

var category = Rng.Next(10);
chars[i] = category switch
{
0 => (char)Rng.Next(1, 32), // control chars (skip null)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Adversarial tests should ideally include null bytes (\0), as they are a common source of vulnerabilities and crashes in string handling, database drivers, and serialization logic. Since the PR summary mentions exercising null bytes, they should be included in this randomized sweep as well.

                0 => (char)Rng.Next(0, 32),               // control chars (including null)

Comment on lines +60 to +63
// ─────────────────────── Board creation sweep ───────────────────────

[Fact]
public async Task BoardCreation_100RandomPayloads_Never500()
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The method RandomGuidString is currently unused in this test file. To maintain code cleanliness, it should be removed if there are no immediate plans to use it.

Guid.NewGuid(),
TriageRunId: Guid.NewGuid(),
ProposalId: Guid.NewGuid(),
PromptVersion: pv,
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using DateTimeOffset.UtcNow inside an FsCheck generator makes the property test non-deterministic. If a failure occurs, the reproduction of that specific failure using the same seed might fail because the timestamp will have changed. It is better to generate the DateTimeOffset using FsCheck's built-in generators to ensure full reproducibility.

Comment on lines +352 to +382
{
case "board":
var board = new Board("Test");
initialUpdatedAt = board.UpdatedAt;
Thread.Sleep(1);
board.Update(name: "Updated");
afterUpdatedAt = board.UpdatedAt;
break;
case "card":
var card = new Card(Guid.NewGuid(), Guid.NewGuid(), "Title");
initialUpdatedAt = card.UpdatedAt;
Thread.Sleep(1);
card.Update(description: "Updated");
afterUpdatedAt = card.UpdatedAt;
break;
case "column":
var col = new Column(Guid.NewGuid(), "Col", 0);
initialUpdatedAt = col.UpdatedAt;
Thread.Sleep(1);
col.SetPosition(1);
afterUpdatedAt = col.UpdatedAt;
break;
case "label":
var label = new Label(Guid.NewGuid(), "Label", "#FF0000");
initialUpdatedAt = label.UpdatedAt;
Thread.Sleep(1);
label.Update(name: "Updated");
afterUpdatedAt = label.UpdatedAt;
break;
default:
throw new InvalidOperationException($"Unknown entity type: {entityType}");
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The use of Thread.Sleep(1) inside a property-based test (which runs 200 iterations by default) significantly slows down the test suite. Furthermore, DateTimeOffset.UtcNow resolution on some operating systems can be as high as 15ms, so a 1ms sleep does not guarantee that the timestamp will advance. Since the assertion on line 385 uses BeOnOrAfter, the test will pass even if the timestamps are identical. Removing the sleeps will improve performance without affecting the validity of the current assertion.

                switch (entityType)
                {
                    case "board":
                        var board = new Board("Test");
                        initialUpdatedAt = board.UpdatedAt;
                        board.Update(name: "Updated");
                        afterUpdatedAt = board.UpdatedAt;
                        break;
                    case "card":
                        var card = new Card(Guid.NewGuid(), Guid.NewGuid(), "Title");
                        initialUpdatedAt = card.UpdatedAt;
                        card.Update(description: "Updated");
                        afterUpdatedAt = card.UpdatedAt;
                        break;
                    case "column":
                        var col = new Column(Guid.NewGuid(), "Col", 0);
                        initialUpdatedAt = col.UpdatedAt;
                        col.SetPosition(1);
                        afterUpdatedAt = col.UpdatedAt;
                        break;
                    case "label":
                        var label = new Label(Guid.NewGuid(), "Label", "#FF0000");
                        initialUpdatedAt = label.UpdatedAt;
                        label.Update(name: "Updated");
                        afterUpdatedAt = label.UpdatedAt;
                        break;
                    default:
                        throw new InvalidOperationException($"Unknown entity type: {entityType}");
                }

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 86e2b02c1e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

throw new InvalidOperationException($"Unknown entity type: {entityType}");
}

afterUpdatedAt.Should().BeOnOrAfter(initialUpdatedAt);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Require UpdatedAt to move forward after entity updates

This property claims to verify that updates advance UpdatedAt, but afterUpdatedAt.Should().BeOnOrAfter(initialUpdatedAt) also passes when timestamps are equal. If a regression stops touching UpdatedAt in Board.Update/Card.Update/Column.SetPosition/Label.Update, this test still passes with afterUpdatedAt == initialUpdatedAt, so it cannot catch the intended failure mode.

Useful? React with 👍 / 👎.

Comment on lines +85 to +89
try {
JSON.parse(param)
} catch {
// Expected - the frontend should handle this gracefully
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove swallowed parse errors from malformed-params test

The assertion is ineffective because JSON.parse(param) is wrapped in an inner try/catch that suppresses all errors before expect(...).not.toThrow() runs. That means this test passes regardless of malformed input behavior and cannot detect regressions in how malformed operation parameters are handled.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds broad property-based and adversarial input testing across backend API/domain/application and frontend to ensure hostile/malformed inputs don’t crash the system (especially avoiding 500s), and to validate serialization/query boundary behavior.

Changes:

  • Added new frontend fast-check adversarial/property tests for proposal/capture/webhook URL/numeric edge cases.
  • Added backend FsCheck/Xunit property tests for capture provenance/payload serialization round-trips and domain/query boundary values.
  • Added backend API adversarial tests and a randomized “no 500s” meta-sweep across multiple endpoints.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
frontend/taskdeck-web/src/tests/property/adversarialData.spec.ts New frontend property/adversarial test suite for JSON round-trips, URL/text/numeric edge cases
backend/tests/Taskdeck.Domain.Tests/PropertyBased/QueryBoundaryValueTests.cs New domain boundary-value tests for GUID/DateTime/string/position/WIP limits + UpdatedAt behavior
backend/tests/Taskdeck.Application.Tests/Fuzz/CaptureProvenanceRoundTripFuzzTests.cs New FsCheck round-trip and deserialization-safety fuzz tests for capture payload/provenance DTOs
backend/tests/Taskdeck.Api.Tests/WebhookUrlAdversarialTests.cs New API adversarial tests for webhook URL validation and robustness (no 500s)
backend/tests/Taskdeck.Api.Tests/ProposalOperationsAdversarialTests.cs New API adversarial tests for proposal creation with malformed/hostile operation payloads
backend/tests/Taskdeck.Api.Tests/No500sMetaTest.cs New randomized cross-endpoint sweep intended to ensure “no 500s” / no stack-trace leakage
backend/tests/Taskdeck.Api.Tests/CaptureAdversarialTests.cs New API adversarial tests for capture endpoint with binary/control/unicode/injection/malformed JSON inputs
Comments suppressed due to low confidence (1)

backend/tests/Taskdeck.Api.Tests/No500sMetaTest.cs:265

  • Despite the class name/summary (“No500s”), this test will still pass if an endpoint returns 500 as long as the body doesn’t contain the checked substrings. If the intent is truly “never 500”, add an assertion that StatusCode < 500; otherwise consider renaming the test/class to reflect that it’s specifically a “no stack trace leakage” check.
    [InlineData("GET", "/api/notifications")]
    public async Task AuthenticatedEndpoints_NeverReturn500WithStackTrace(string method, string path)
    {
        await EnsureAuthenticatedAsync();

        HttpResponseMessage response;
        if (method == "GET")
        {
            response = await _client.GetAsync(path);
        }
        else
        {
            response = await _client.PostAsync(path,
                new StringContent("{}", Encoding.UTF8, "application/json"));
        }

        if ((int)response.StatusCode >= 500)
        {
            var body = await response.Content.ReadAsStringAsync();
            body.Should().NotContain("StackTrace",
                $"{method} {path} returned 500 with stack trace exposed");
            body.Should().NotContain("at Taskdeck.",
                $"{method} {path} returned 500 with internal exception details exposed");
        }
    }
}


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +71 to +91
it('malformed operation parameters string does not throw when parsed cautiously', () => {
const malformedParams = [
'',
'not json',
'{',
'[',
'{"unclosed": ',
'null',
'12345',
'<xml>data</xml>',
]

for (const param of malformedParams) {
expect(() => {
try {
JSON.parse(param)
} catch {
// Expected - the frontend should handle this gracefully
}
}).not.toThrow()
}
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is effectively a no-op: JSON.parse errors are caught and swallowed inside the expectation, so the outer expect().not.toThrow() will always pass regardless of input. Consider exercising the actual parsing/validation helper the app uses (or add one) and assert a concrete outcome (e.g., returns null/Result/error state) for malformed parameter strings.

Copilot uses AI. Check for mistakes.
Comment on lines +220 to +238
it('capture text with binary-like data should serialize', () => {
// Simulate various binary-like string patterns
const binaryPatterns = [
'\x00\x01\x02\x03\x04\x05',
String.fromCharCode(...Array.from({ length: 256 }, (_, i) => i)),
'\uFFFE\uFFFF', // non-characters
'\uD800', // lone surrogate (will be replaced in JSON)
]

for (const pattern of binaryPatterns) {
expect(() => {
try {
const json = JSON.stringify({ text: pattern })
JSON.parse(json)
} catch {
// Some patterns may fail JSON serialization, which is expected
}
}).not.toThrow()
}
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue as above: the try/catch inside expect().not.toThrow() swallows any JSON.stringify/JSON.parse failures, so this assertion will always pass and doesn't validate behavior. If the goal is resilience, assert that serialization succeeds for the supported patterns, and explicitly assert failure/handling behavior for patterns you expect to be rejected.

Copilot uses AI. Check for mistakes.
Comment on lines +241 to +251
it('very long capture text (50K chars) should not freeze', () => {
const longText = 'x'.repeat(50_000)
const start = performance.now()

const json = JSON.stringify({ text: longText })
JSON.parse(json)

const elapsed = performance.now() - start
// Should complete well under 1 second
expect(elapsed).toBeLessThan(1000)
})
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This uses a hard 1s wall-clock threshold (performance.now()) which is prone to CI flakiness under load and doesn't really measure UI “freeze” (it only measures JSON stringify/parse). Consider removing the timing assertion (just ensure it completes), or make it much more tolerant and/or run it as a benchmark-style test outside the main suite.

Copilot uses AI. Check for mistakes.
Comment on lines +135 to +140
public void Card_DueDate_AcceptsNull()
{
var card = new Card(Guid.NewGuid(), Guid.NewGuid(), "Title");
card.Update(dueDate: null);
card.DueDate.Should().BeNull();
}
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test doesn’t actually verify that a due date can be cleared: the card starts with DueDate == null, and Card.Update(dueDate: null) is a no-op with the current Card.Update implementation (it only assigns when dueDate.HasValue). To test “accepts null/clear”, first set a non-null due date, then call Update(dueDate: null) and assert it becomes null (or rename the test if clearing isn’t supported).

Copilot uses AI. Check for mistakes.
Comment on lines +80 to +89
{
var body = await response.Content.ReadAsStringAsync();
body.Should().NotContain("System.", "500 response should not contain stack traces");
body.Should().NotContain("at Taskdeck.", "500 response should not contain stack traces");
}
}
}

// ─────────────────────── Capture creation sweep ───────────────────────

Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stack-trace body check is unreachable: the BeLessThan(500) assertion will already fail the test before the subsequent if (StatusCode >= 500) block can run. If you want to assert both “not 500” and “no stack trace if 500”, capture the body first and/or use an AssertionScope so both checks can be evaluated.

Copilot uses AI. Check for mistakes.
Comment on lines +108 to +116
var body = await response.Content.ReadAsStringAsync();
body.Should().NotContain("System.", "500 response should not contain stack traces");
}
}
}

// ─────────────────────── Card creation sweep ───────────────────────

[Fact]
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same unreachable-code issue here: the BeLessThan(500) assertion will throw before the if (StatusCode >= 500) branch can execute, so the response-body redaction check never runs. Consider restructuring to always read/assert on the body when status >= 500 (or use AssertionScope).

Copilot uses AI. Check for mistakes.
Comment on lines +37 to +58
private static readonly Random Rng = new(42); // deterministic seed for reproducibility

private static string RandomString(int minLen, int maxLen)
{
var length = Rng.Next(minLen, maxLen + 1);
var chars = new char[length];
for (int i = 0; i < length; i++)
{
// Mix of ASCII, unicode, and control chars
var category = Rng.Next(10);
chars[i] = category switch
{
0 => (char)Rng.Next(1, 32), // control chars (skip null)
1 => (char)Rng.Next(0x4E00, 0x4F00), // CJK
2 => (char)Rng.Next(0x0600, 0x0700), // Arabic
3 => (char)Rng.Next(0x0300, 0x0370), // combining diacriticals
4 => (char)Rng.Next(0x2000, 0x2070), // general punctuation / special
_ => (char)Rng.Next(0x20, 0x7F), // printable ASCII
};
}
return new string(chars);
}
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rng is a shared static Random consumed across multiple [Fact]s. Because xUnit does not guarantee test execution order, the sequence of “random” values (and therefore failures) can change between runs despite the fixed seed. To keep runs reproducible, instantiate a new Random(42) per test (or reset the RNG at the start of each test), rather than sharing a single static instance.

Copilot uses AI. Check for mistakes.
Comment on lines +10 to +108
/// <summary>
/// Adversarial tests for webhook endpoint URL validation.
/// Verifies that dangerous, malformed, and RFC 3986 edge-case URLs
/// are rejected with 4xx — never 5xx.
/// </summary>
public class WebhookUrlAdversarialTests : IClassFixture<TestWebApplicationFactory>
{
private readonly TestWebApplicationFactory _factory;
private readonly HttpClient _client;
private bool _isAuthenticated;
private Guid? _boardId;

public WebhookUrlAdversarialTests(TestWebApplicationFactory factory)
{
_factory = factory;
_client = factory.CreateClient();
}

private async Task EnsureAuthenticatedAsync()
{
if (_isAuthenticated) return;
await ApiTestHarness.AuthenticateAsync(_client, "webhook-adversarial");
_isAuthenticated = true;
}

private async Task<Guid> EnsureBoardAsync()
{
if (_boardId.HasValue) return _boardId.Value;
await EnsureAuthenticatedAsync();
var board = await ApiTestHarness.CreateBoardAsync(_client, "webhook-test");
_boardId = board.Id;
return board.Id;
}

// ─────────────────────── RFC 3986 edge-case URLs ───────────────────────

public static IEnumerable<object[]> AdversarialWebhookUrls()
{
// Dangerous URL schemes
yield return new object[] { "javascript:alert(1)" };
yield return new object[] { "data:text/html,<script>alert(1)</script>" };
yield return new object[] { "data:text/html;base64,PHNjcmlwdD5hbGVydCgxKTwvc2NyaXB0Pg==" };
yield return new object[] { "vbscript:MsgBox(1)" };
yield return new object[] { "file:///etc/passwd" };
yield return new object[] { "ftp://evil.com/payload" };

// URLs with credentials (userinfo component)
yield return new object[] { "https://admin:password@evil.com/webhook" };
yield return new object[] { "http://user:pass@127.0.0.1/callback" };

// Internal/localhost URLs (SSRF vectors)
yield return new object[] { "http://localhost/webhook" };
yield return new object[] { "http://127.0.0.1/webhook" };
yield return new object[] { "http://[::1]/webhook" };
yield return new object[] { "http://0.0.0.0/webhook" };
yield return new object[] { "http://169.254.169.254/latest/meta-data/" };
yield return new object[] { "http://metadata.google.internal/" };

// Malformed URLs
yield return new object[] { "" };
yield return new object[] { " " };
yield return new object[] { "not-a-url" };
yield return new object[] { "://missing-scheme" };
yield return new object[] { "http://" };
yield return new object[] { "http:///no-host" };

// URLs with injection payloads
yield return new object[] { "https://evil.com/webhook?q='; DROP TABLE webhooks; --" };
yield return new object[] { "https://evil.com/webhook#<script>alert(1)</script>" };
yield return new object[] { "https://evil.com/\u0000null-byte" };
yield return new object[] { "https://evil.com/\r\nHeader-Injection: true" };

// URLs with unicode
yield return new object[] { "https://evil.com/\u200Bhidden" };
yield return new object[] { "https://evil.com/\u202Efdp.exe" };
yield return new object[] { "https://xn--n3h.com/webhook" }; // punycode emoji domain

// Extremely long URL
yield return new object[] { "https://example.com/" + new string('a', 10_000) };

// URL with port boundaries
yield return new object[] { "https://example.com:0/webhook" };
yield return new object[] { "https://example.com:99999/webhook" };
yield return new object[] { "https://example.com:-1/webhook" };
}

[Theory]
[MemberData(nameof(AdversarialWebhookUrls))]
public async Task CreateWebhook_WithAdversarialUrl_NeverReturns500(string endpointUrl)
{
var boardId = await EnsureBoardAsync();

var response = await _client.PostAsJsonAsync(
$"/api/boards/{boardId}/webhooks",
new CreateOutboundWebhookSubscriptionDto(endpointUrl));

((int)response.StatusCode).Should().BeLessThan(500,
$"Webhook creation returned 500 for URL: {endpointUrl}");
}
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The XML doc says adversarial URLs “are rejected with 4xx — never 5xx”, but the assertion only checks < 500, so 2xx responses (accepting a dangerous URL) would still pass. If the requirement is rejection, assert a 4xx response (and consider environment-specific exceptions like localhost being allowed in Development per OutboundWebhooksApiTests.cs:104-113).

Copilot uses AI. Check for mistakes.
Comment on lines +110 to +124
[Theory]
[InlineData("https://example.com/webhook")]
[InlineData("https://hooks.slack.com/services/T00/B00/xxxx")]
[InlineData("https://example.com:8443/api/webhook")]
public async Task CreateWebhook_WithValidUrl_Succeeds(string endpointUrl)
{
var boardId = await EnsureBoardAsync();

var response = await _client.PostAsJsonAsync(
$"/api/boards/{boardId}/webhooks",
new CreateOutboundWebhookSubscriptionDto(endpointUrl));

// Valid HTTPS URLs should either succeed or be accepted
((int)response.StatusCode).Should().BeLessThan(500);
}
Copy link

Copilot AI Apr 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test name says “Succeeds”, but the assertion only checks that the response is not a 5xx. Either tighten the assertion to the expected success status (e.g., 201 Created) or rename the test to reflect the weaker contract (“never 500”).

Copilot uses AI. Check for mistakes.
- Remove unnecessary backslash escapes in JSON string constant (no-useless-escape)
- Replace hard wall-clock threshold with functional round-trip assertion
- Replace swallowed-exception tests with explicit throw/value assertions
- Fix binary data test to verify actual round-trip instead of swallowing errors
- Replace static Random with per-test CreateRng() for thread safety
- Add null byte (\0) to adversarial character set
- Make AuthenticatedEndpoints test actually fail on 500 responses
- Remove unreachable stack-trace checks after BeLessThan assertion
- Eliminates Thread.Sleep(1) * 200 iterations performance cost
- Uses BeAfter instead of BeOnOrAfter for strict timestamp advancement
- Uses 16ms sleep to exceed Windows timer resolution
@Chris0Jeky
Copy link
Copy Markdown
Owner Author

Review findings and fixes

Investigated CI failures and all bot review comments. All issues addressed in follow-up commits.

CI fix (Priority 1)

  • ESLint no-useless-escape on line 26: Unnecessary backslash escapes in adversarialData.spec.ts JSON string constant. Fixed by using single quotes around the JSON string instead of escaping inner double quotes.

Bot review issues addressed

# Source File Fix
1 Codex P1 No500sMetaTest:252 AuthenticatedEndpoints_NeverReturn500 now asserts < 500 directly instead of only inspecting body on 500
2 Codex P2 WebhookUrlAdversarialTests:123 Valid URL test now asserts 201 Created instead of just < 500
3 Codex P2 adversarialData.spec.ts:250 Removed hard wall-clock threshold; replaced with functional round-trip assertion
4 Gemini No500sMetaTest:37 Replaced static Random with per-test CreateRng() factory for thread safety
5 Gemini No500sMetaTest:49 Added null byte (\0) to adversarial character set
6 Gemini No500sMetaTest:63 RandomGuidString was already removed in prior self-review commit
7 Gemini CaptureProvenanceRoundTripFuzzTests:276 Replaced DateTimeOffset.UtcNow with fixed timestamp for deterministic FsCheck reproduction
8 Gemini QueryBoundaryValueTests:382 Replaced property-based Touch test (200 iterations * Thread.Sleep) with 4 individual Facts using 16ms sleep
9 Codex P2 QueryBoundaryValueTests:385 Changed BeOnOrAfter to BeAfter for strict timestamp advancement
10 Codex P2 adversarialData.spec.ts:89 Replaced swallowed-exception test with explicit throw/value assertions per input

Fresh adversarial review findings (also fixed)

  • Unreachable stack-trace checks: In BoardCreation_100RandomPayloads_Never500 and CaptureCreation_100RandomPayloads_Never500, the if (StatusCode >= 500) blocks were unreachable after BeLessThan(500) assertion. Removed the dead code.
  • Card_DueDate_AcceptsNull was a no-op: The test passed null to Update(dueDate: null) on a card that already had null DueDate. Fixed to verify that null is a no-op that preserves an existing due date.
  • Binary data test swallowed exceptions: The capture text with binary-like data test used try/catch inside expect().not.toThrow(), making it always pass. Fixed to assert actual round-trip correctness.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 746bbc880f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +106 to +107
((int)response.StatusCode).Should().BeLessThan(500,
$"Webhook creation returned 500 for URL: {endpointUrl}");
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Assert 4xx rejection for adversarial webhook endpoints

CreateWebhook_WithAdversarialUrl_NeverReturns500 currently only asserts status < 500, so a regression that starts accepting dangerous endpoints (for example javascript:/data:/metadata-host URLs) with 2xx would still pass this test. That makes this adversarial security test a false negative for URL-validation/SSRF regressions; it should assert explicit rejection semantics (4xx), with any intentionally allowed cases split into separate tests.

Useful? React with 👍 / 👎.

@Chris0Jeky Chris0Jeky merged commit ce8c5be into main Apr 12, 2026
24 checks passed
@Chris0Jeky Chris0Jeky deleted the test/property-based-adversarial-input branch April 12, 2026 01:06
@github-project-automation github-project-automation bot moved this from Pending to Done in Taskdeck Execution Apr 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

TST-50: Stress the boundaries — property-based and adversarial input tests

2 participants